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Abstract 


The objective of this work was to identify and estimate complexity and risks associated with the 
development and testing of new low-cost medium-scale X-plane aircraft primarily focused on air 
transport operations. Piloting modes that were evaluated for this task were manned, remotely piloted, and 
unmanned flight research programs. This analysis was conducted early in the data collection period for 
X-plane concept vehicles before preliminary designs were complete. Over 50 different aircraft and system 
topics were used to evaluate the three piloting control modes. Expert group evaluations from a diverse set 
of pilots, engineers, and other experts at Aeronautics Research Mission Directorate centers within the 
National Aeronautics and Space Administration provided qualitative reasoning on the many issues 
surrounding the decisions regarding piloting modes. The group evaluations were numerically rated to 
evaluate each topic quantitatively and were used to provide independent criteria for vehicle complexity 
and risk. An Edwards Air Force Base instruction document was identified that emerged as a source of the 
effects found in our qualitative and quantitative data. The study showed that a manned aircraft was the 
best choice to align with test activities for transport aircraft flight research from a low-complexity and 
low-risk perspective. The study concluded that a manned aircraft option would minimize the risk and 
complexity to improve flight-test efficiency and bound the cost of the flight-test portion of the program. 
Several key findings and discriminators between the three modes are discussed in detail. 


Nomenclature 
6-DOF six degrees-of-freedom 
AFRC Armstrong Flight Research Center 
ARMD Aeronautics Research Mission Directorate 
ATC air traffic control 
BLI boundary layer ingestion 
CAS Convergent Aeronautics Solutions 
DE development engineering 
DFCS digital flight control system 
EAFB Edwards Air Force Base 
FAA Federal Aviation Administration 
F-TS flight-test support 
GVT ground vibration test 
HILS hardware-in-the-loop simulation 
NASA National Aeronautics and Space Administration 
NRC National Research Council 
OR other requirements 
RC research capability 
RPV remotely piloted vehicle 
TACP Transformative Aeronautics Concepts Program 
UAS unmanned aircraft system 
VR vehicle requirements 
VS vehicle subsystems 

Introduction 


The objective of this work was to provide a qualitative exploration of complexity and risk topics 
differentiating manned, remotely piloted, and unmanned X-plane concepts, and the operational challenges 
associated with each piloting control mode. This investigation was one of the first studies to be funded 


under a subproject in the Aeronautics Research Mission Directorate’s (ARMD) new Convergent 
Aeronautics Solutions (CAS) project under the Transformative Aeronautics Concepts Program (TACP), 
which focuses on convergent solutions for future aeronautics research. This analysis was conducted 
before the ARMD’s Spring 2016 announcement of a formal X-plane New Aviation Horizons Initiative 
(ref. 1). Early task documentation encouraged an investigation of potential cost effective flight-test 
approaches to meet future ARMD flight research needs. The early low-cost objective of the medium scale 
X-plane was $50 million to $100 million to build and complete the first flight. Two near-term goals were 
also identified: 1) perform trade studies for vehicle configuration and concepts of operation and 
2) develop vehicle research objectives and requirements. A National Research Council (NRC) report from 
2012 also recommended that “NASA should also develop cost effective flight research vehicles to 
demonstrate innovative aerospace technology in flight.” (ref. 2). Ultimately, this work is intended to 
provide decision support data to estimate costs and risks associated with building and testing medium- 
sized X-plane aircraft, approximately the size of a Gulfstream Aerospace (Savannah, Georgia) GIII 
aircraft or larger. This report describes the final approach chosen for the summary process, and provides 
some discussion of several interesting findings. 

Three distinct modes of aircraft operation were compared during this study, and are defined as 
follows. The manned pilot control mode is defined as having at least one pilot on board the aircraft. The 
remotely piloted vehicle (RPV) control mode is defined as having at least one pilot in a ground control 
station that actively flies the aircraft in real time with no personnel on board the aircraft at any time. The 
autonomous pilot control mode is defined as having a preprogrammed flight path with control waypoints 
and at least one pilot at a ground control station actively monitoring the flight with the ability to alter the 
flight path for testing or emergency purposes. The autonomous pilot control mode also has no personnel 
on board the aircraft at any time. 

The general framework of quantitative analysis (ref. 3) was used to support evaluation of topics or 
themes. These topics and themes were identified and then assessed by an expert panel. The qualitative 
data alone are based on non-quantifiable topics surrounding research capabilities and vehicle 
requirements. A qualitative technique identifies, through subjective judgment, the important effects or 
influences of each proposed pilot control mode. The expert panel was composed of 22 members including 
manned and unmanned aircraft system (UAS) pilots, dynamics and control engineers, aeronautics systems 
analysts, operations engineers with UAS program experience, a UAS chief engineer, a cost estimator, 
aerodynamics engineers, a modeling and simulation engineer, and an instrumentation engineer. This 
group had a combined experience of over 400 man-years of aircraft experience. Due to the limited 
availablity of experts, time, and resources, the purposive/strategic sampling principle was used (ref. 3). 
The expert panel qualitatively identified, compared, and ranked a total of 54 different topic items using a 
simple quantitative scoring procedure. Additionally, the following quantitative method was used to help 
provide additional insight. A simplified rating system was developed using vehicle complexity and risk 
criteria. Numerical values from 0 to 5 were assigned to each topic, where a score of 0 indicated non- 
challenging influences, while a score of 5 indicated problematic influences. 

Vehicle complexity encompasses a combination of the estimated schedule and costs. Risk criteria 
were a combination of both vehicle risk and mission risk. Vehicle risk and crew risk were assumed to 
have similar program consequences and were combined for the analysis. Some examples of vehicle risk 
resulting in loss of vehicle included loss of control and loss of situational awareness. Some examples of 
mission risk included immature technologies, integration difficulties, inability to collect research data, 
simulation limitations, and unknown unknowns. 

Qualitative results of this study contain the major discussion points and high-level conclusions, while 
the quantitative results provide an estimate of the overall influence of the topic items. Both qualitative and 
quantitative results strongly favored a manned aircraft for transport aircraft flight physics research, based 
on estimated risk and complexity. 


Candidate Aircraft Flight Research Goals 


A total of ten candidate aircraft configurations were initially considered by the CAS subproject for 
clean and efficient air transport X-planes. These ten configurations covered a broad spectrum of both 
classic aircraft configurations and more aerodynamically or structurally advanced configurations. Each 
configuration was considered for a large-scale flight research campaign similar to those of the X-29 
(Grumman Aerospace Corporation, now Northrop Grumman Corporation, West Falls Church, Virginia) 
and X-31 aircraft (Rockwell-Messerschmitt-Bé6lkow-Blohm, United States / Germany), with about 100 to 
150 flights. The configuration selection process was performed in parallel with this study, and focused on 
structural, aerodynamic, and propulsive issues, and the value of the flight research that each configuration 
could produce. The list of flight research goals and aircraft engineering issues (not prioritized and 
incomplete) was defined in early documentation and included: 

e Full-scale dynamics, handling qualities, and pilot workload impacts, 

e Boundary layer ingestion (BLI) propulsion system performance, 

e Test beds for multiple aerodynamic and propulsive technologies, 

e Low-speed stability and control performance, 

e Terminal area noise and acoustics analysis validation (engine shielding and airframe noise), 
e Flight controls for tailless vehicles, 

e Unique pressurization geometries, 

e Efficient transonic performance, 

e System integration, 

e Quantifiable structural efficiency gains. 


Expected Vehicle Specifications 


A wide range of vehicle factors made this evaluation difficult to bound and complicated the 
formulation of objective comparisons. Candidate vehicle physical configurations varied significantly in 
layout, size, and weight as determined by specific fight research objectives. Each aircraft was assumed to 
be about the same size as a Gulfstream GIII aircraft or larger. Approximate values assumed for the 
bounding physical configuration parameters were expected to exist in the following ranges: 

e Scale: 40 to 65 percent of full scale, 

e Wingspan: 55 ft to 75 ft, 

e Empty weight: 14,000 Ib to 40,000 Ib, 

e Primary Flight Condition: High subsonic to low transonic at altitudes from 20,000 ft to 

35,000 ft MSL. 

When combined with the flight-test goals listed in the Candidate Aircraft Flight Research Goals 
section, these anticipated X-plane parameters yielded aircraft that would require testing at a large flight- 
test range. The most likely test location was identified as Edwards Air Force Base (EAFB) / National 
Aeronautics and Space Administration (NASA) Armstrong Flight Research Center (AFRC) (Edwards, 
California). 

A detailed manned vs unmanned analysis of each concept vehicle with respect to the research and 
specifications could not be performed at the early stages of the analysis, as many of the configurations 
were conceptual at best. The scope of this study focused on the research and vehicle specifications from a 
more general perspective. A truly rigorous analysis and comparison would depend on the specifics of 
each vehicle configuration, requiring information on the level of a conceptual design for each. Given the 
wide variations in both the research goals and configurations of the ten candidate aircraft, this study 
focused instead on identifying complexity and risk characteristics that were thought to be common to any 
decisions regarding a future X-plane. 


Range Limitations and Impact of Sense & Avoid Requirements 


A small unmanned aircraft system (UAS) work area is available on the EAFB range. Autonomous 
and RPV vehicles that use the UAS work area can take off and land from the lakebed runways, and 
operations can be conducted on an unproven vehicle without significant impacts. Other EAFB aircraft are 
unaffected by unproven UAS operating in the UAS work area because the traffic patterns easily avoid the 
UAS work area. The work area, however, is constrained to RPV or autonomous vehicles with ceilings of 
10,000 ft or less travelling slower than 120 knots. At these maximum states, test conditions can only be 
held for short periods of time (typically about a few minutes in duration). The UAS work area would be 
too small for the aircraft and research objectives that were considered in this study. 

Due to the task requirements of the target X-plane specification, the full range at EAFB would need 
to be utilized. The medium-sized UAS X-plane aircraft was projected to detect factor traffic cooperativly 
and would be unable to react to other air traffic quickly. In portions of the EAFB range the test altitude is 
unlimited, and in other portions the test ceiling can vary from 29,000 ft to 50,000 ft. Test vehicles using 
the full range are subject to much stricter procedures and must be able to meet numerous operating 
constraints for safety and traffic avoidance. These limitations and constraints are documented in EAFB 
Instruction 13-1004 which governs UAS flying and airfield operations. 

UAS vehicles operating at EAFB are currently separated into five categories defined by their sense 
and avoid capabilities. The sense and avoid capability of the UAS is a risk reduction metric that impacts 
the airspace management of the test vehicle and the efficiency of flight-test operations. Chapter 14 of 
EAFB Instruction 13-1004 governs UAS flying and airfield operations for UAS Operations. The items 
below (reprinted verbatim) provide a sense of the contraints on UAS operations and necessary equipment 
in the area of air traffic control (ATC) (ref. 4). 

5 UAS type definitions segregate vehicles by sense and avoid and deviation capabilities: 

e 14.4.1. Type 1: UAS has the ability to conduct sense and avoid to an equivalent level of capability 

as a manned aircraft (cooperative and non-cooperative traffic). 

e 14.4.2. Type 2: UAS able to detect factor traffic (cooperative only) and take appropriate avoidance 
action in a timely manner (usually within a few seconds). The detection to action decision loop 
only involves the UAS and the operator. 

e 14.4.3. Type 3: UAS able to detect factor traffic (cooperative only), but unable to react in a timely 
manner (usually within a few seconds). This delay may be due to detection method (ATC traffic 
monitoring, Chase aircraft) and/or latency inherent in UAS system (long link delays, complicated 
command sequences). 

e 14.4.4. Type 4: UAS unable to deviate from flight path for traffic avoidance. ATC may be able to 
detect the conflict and direct the conflicting traffic to maneuver (ATC transponder required). 

e 14.4.5. Type 5: UAS unable to deviate from flight path for traffic avoidance and ATC unable to 
accurately track a UAS to detect traffic conflicts (no transponder). 

From the possible aircraft configurations under program consideration, “Type 3” was identified as the 
most probable designation. UAS risk mitigation requirements and procedures can also be found in 
Chapter 14 of EAFB Instruction 113-1004 as well as RCC Document 323-99 (ref. 5). A mitigation matrix 
provided therein outlines a general process that progresses risk mitigation from unproven (lowest level), 
through experimental (expected highest level for a medium-scale X-plane) to provisional and mature 
systems. An example of the required risk mitigation procedures for the unproven class of UAS is 
presented in table 1. 


Table 1. Common unproven class UAS mitigation requirements. 


Procedures 
Exclusive use airspace 
Sanitized ground footprint 
Lakebed or exclusion zone takeoff and landing 
Chase aircraft to provide see and avoid 
Flight termination system 
Road closure 
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The RCC document 323-99 (ref. 5) outlines range safety for unmanned air vehicles, which provides 
“criteria based on guidance from safety specialists, existing reference standards and policies, and 
established procedures.” This document establishes guidance for restricted airspace ranges and verifies 
compliance to National Airspace Federal Aviation Administration (FAA) requirements outside of 
restricted airspace. The document details the above criteria and indicates system maturity and system 
reliability serve as assurances that a UAS is safe to operate and will not pose additional risk to other 
vehicles, property, or human life. These assurances alone require significant restricted airspace operation 
to establish confidence in each airframe. 


Analysis Objectives and Assumptions 


An objective of this work was to provide decision support data and comparisons of the perceived 
risks of large-scale X-plane concept vehicles with traceability to rationale and recommendations. Several 
separate studies done in parallel with this process showed a strong correlation between the cost and risk of 
building an airframe and the empty weight of the aircraft (refs. 6 and 7). Additionally, it was not expected 
that any high technical risk subsystems necessary for specific research objectives would be included in 
the airframe. The analysis was conducted assuming little technical risk up to the delivery of the airframe. 
In general, this study focused on the complexity and risk topics most prominent during the flight-testing 
portion of an X-plane program. 


Qualitative Analysis Objectives 


The qualitative analysis (ref. 3) was expected to provide valuable information gained from the 
experience of flight-test engineers and staff. Detailed notes were kept to capture data used for the 
qualitative analysis. The data were used to provide a general understanding and to clarify and capture the 
complexities and risks associated with high level tradeoffs between manned and unmanned vehicles. 
Additionally, it was anticipated that the qualitative analysis would highlight several topics with strong 
supporting evaluation rationale. 


Quantitative Analysis Objectives 


The quantitative analysis was designed to include a general rating system intended to identify 
differences and similarities between each of the manned, remotely piloted, and unmanned X-plane 
piloting modes. The rating criteria for topic is outlined in table 2. The authors attempted to use a time 
weighting criteria in addition to the analysis presented, but found that the time weighting had very little 
effect on the results and decided not to include the description or results. An expectation of the 
quantitative analysis was that it would identify impacts for each pilot control mode, which would in turn 
provide an overall result for the criteria as a whole. 


Table 2. Topic rating criteria. 


Rating Rating description 
value (Technical, schedule, and cost) 
B) Very complex topic that was technical and/or contributed largely to cost and schedule 


4 Complex topic that was technical and/or contributed to cost and schedule 

3 Moderately complex topic that was somewhat technical and/or contributed to cost and schedule 

2 Mildly complex topic that was not very technical nor a considerable contribution to cost and schedule 
1 A requirement but not difficult to accomplish 

0 Not a requirement or not applicable 


Analysis Assumptions 


The X-plane concept vehicles were assumed to be subsonic medium-scale aircraft. Experience with 
testing this class of aircraft indicated that ejection seats would likely not be implimented in manned 
vehicles, and predetermined egress procedures could be executed in most emergency cases. In a program 
of this scale it was expected that the loss of vehicle would be equally detrimental in terms of risk as loss 
of a pilot; it was assumed that either would cause the program to be canceled. These assumptions are 
summarized in table 3. 


Table 3. Basic assumptions used for evaluation. 


Assumption Manned RPV Autonomous 
1 No ejection seat will be implemented on the test vehicle v N/A N/A 
2 Generalized evaluation of a medium-sized aircraft configuration v v v 
3 Risk to pilot and vehicle were not independently considered v N/A N/A 
4 | Vehicle will be statically stable v v v 
5 Vehicle will operate subsonic and possibly transonic v v v 
6 | Vehicle will be flown in the Edwards AFB range v v v 
7 | Autonomous operation will not be a research objective v v v 
The topic categories and data for this analysis were generated during multiple discussions with flight- 
test experts and pilots of both manned and unmanned aircraft at ARMD centers within NASA. In general, 


these topics and data were qualitative in nature regarding flight-testing functions. Table 4a presents the 
general themes and the distribution of complexity and risk topics within each theme. Table 4b lists the 
specific 54 complexity and risk topics that were considered. Several of these topics were evaluated from 
different perspectives that included vehicle requirements (VR), research capability (RC), vehicle 
subsystems (VS), developmental engineering (DE), flight-test support (F-TS) and other requirements 
(OR). Several topics did not deviate over vehicle type. Some examples of these topics were transponders, 
radio communication, engine performance, and chase aircraft. 


Table 4a. Complexity and risk themes. 


Themes 
. Vehicle requirements (VR) Research capability (RC) Other requirements (OR) 
Eepiety 22 topics 9 topics 17 Topics 
Risk Vehicle subsystems (VS) Developmental engineering (DE) Flight-test support (F-TS) 
20 topics 12 topics 13 topics 


Table 4b. Complexity and risk topics. 


Topics 


Airframe noise (RC) 


Flight envelope restrictions (OR) 


Mission timeline (OR) (F-TS) 


Airframe performance (RC) (DE) 


Flight instrumentation and data 
recording (VR) (VS) 


Navigation (VR) (VS) 


Airspace availability (VS) 


Flight termination system 


Operations workforce - physical danger 


(VR) (VS) (OR) (F-TS) 
Airworthiness process Pilot display instrumentation (VR) Operations workforce - program 
(OR) (F-TS) (VS) (OR) (F-TS) 
Autopilot / auto throttle / auto land ; os 
(VR) (VS) Flutter (RC) (DE) Pilot situational awareness (VS) 


Chase aircraft (OR) (F-TS) 


Frequency requirements 
(OR) (F-TS) 


Pilot workload quantification (RC) 


Cockpit / ground control station 


Ground operation (OR) 


Power requirements (VR) (VS) 


(VR) (VS) 
Command and control link Ground tests - preflight, GVT, 
(VR) (VS) et cetera (OR) (DE) Radar {V') 
Control law development Ground tests ‘ ae 
(RC) (DE) - taxi, et cetera (OR) (F-TS) Radio communication (VR) (VS) 
Control room (OR) Handling qualities (RC) (DE) Range restrictions (OR) (F-TS) 


Egress (VR) (VS) 


Instrumentation ground testing 
(F-TS) 


Sense and avoid - airspace availability (VR) 


Engineering workforce - 
developmental (OR) (DE) 


Laminar flow (RC) (DE) 


Sense and avoid - pilot SA (VS) 


Engineering workforce 
- test support (OR) (F-TS) 


Line of sight (VR) (VS) 


Simulation - HILS (OR) (DE) 


Engine performance 


Lost link - mitigation 


Simulation - 6-DOF (OR) (DE) 


(RC) (DE) (VR) (VS) 
Environmental control system Lost link - self destruct Stability and control — development 
(VR) (VS) (VR) (DE) (F-TS) 
Envelope expansion Mission management Stability and control - evaluation 
(OR) (F-TS) (VR) (VS) (RC) (VR) 
Flight control - conventional see ; ; 
(VR) (VS) Mission planning (OR) Transponder (VR) (VS) 
Flight control - full DFCS in cist ; . 
(VR) (VS) Mission support (OR) (F-TS) Upset recoveries (DE) 


The qualitative data for the complexity and risk topics were collected as experience-based 
observations or considerations that identified impacts and challenges specific to each of the piloting 
options. Scores for individual topics ranged from 0 to 5, with 0 assessed as low and 5 assessed as high 


complexity or risk. 


A large amount of experienced based analytical data were generated by the expert panel that 
identified support rational and challenges for each piloting mode of operation. This section summarizes 
the complexity results starting with the manned option, followed by the remotely piloted option and 


autonomous options. 


Results 


Qualitative Complexity Results 


Manned Vehicle Results Synopsis 


Qualitative material that favored manned vehicles appeared in ground operations, instrumentation, 
and testing procedures. Ground operation restrictions were identified as minimal during flight-testing of 
manned vehicles relative to unmanned vehicles. The ground operations of new vehicles have numerous 
significant impacts due to Air Force directives. The manned vehicle option is largely unaffected by these 
directives. Key characteristics supporting the manned vehicle were: 

e A conventional control system could be implemented with no impact to the program. 

e Aircraft pilot display instrumentation requirements in a manned vehicle do not have to be complex 

to be functional. 

e For manned vehicles, the performance and flying qualities flight-testing procedures and processes 

are well known and would not require additional development. 

Challenges for manned vehicles included cabin pressurization and emergency egress operations. 
Pressurization system requirements are expected to be more complex in manned vehicles as compared to 
unmanned vehicles. Depending on the configuration of the aircraft, emergency egress procedures would 
also be more complex. Although these challenges affect complexity and risk, elimination of each item 
was not expected to have a significant impact on the potential program. Manned vehicle operations are 
well known and regularly practiced, and tend to not increase flight-test complexity. 


Remotely Piloted Vehicle Results Synopsis 


Topics that favored remotely piloted vehicles included modeling, vehicle software assimilation, and 
enabling control law research. Similar software was expected to be used both for simulation and vehicle 
control where each would not be independent efforts, reducing superfluous duplication. Additionally, 
remotely controlled vehicles provide a platform that enables control law research. 

Unfortunately, remotely piloted vehicles inherently encounter several challenging topics specific to 
flight-testing. Situational awareness of a remote piloted vehicle is limited and challenging, increasing the 
difficulty of many standard flight objectives and the complexity of vehicle recovery if needed. Typical 
processes were expected to require lost link risk mitigation with the ability to put the vehicle into a 
holding pattern until the link was restored and the implementation of a flight termination system in 
situations where a vehicle would become or was uncontrollable. These topics, together with uplink and 
downlink requirements, increase complexity both before and during flight-testing. Due to the above 
requirements, a digital flight control system would be highly recommended, further increasing complexity 
throughout the development, validation, and testing. The air and ground testing of unproven (Type 3) 
remotely piloted vehicles is also considerably more restricted than that of manned vehicles. 


Autonomous Vehicle Results Synopsis 


Supporting topics for unmanned autonomous vehicles included extended mission timelines, effective 
loss of signal mitigations, and enabling control law research. Similar to remotely piloted vehicles, 
autonomous vehicles provide a platform that enables control law research. Flight safety requirements for 
unmanned vehicles regarding loss of signal have proven and effective mitigations. Although not expected 
in any flight-testing of the proposed X-planes, another supporting topic was the possibility of extended 
mission timelines (meaning tests longer than 8 hours). 

Challenges that were identified regarding autonomous vehicles were the implementation of a flight 
termination system, uplink and downlink testing, and air and ground testing restrictions. These are the 
same challenges that were identified for the remotely piloted vehicles; however, additional challenges for 
fully autonomous operations were noted. An autonomous vehicle requires a digital flight control system 
that greatly increases complexity, development, validation, and testing. Emergency “non-standard” aborts 
are difficult to anticipate and perform in autonomous vehicles. A significant hurdle to efficient flight 
research was that mission management for fully autonomous operations often imposes multiple lengthy 
and complex preflight review processes. The survey group’s collective experience with medium-sized 


autonomous vehicles indicated that flight-test programs conducted with vehicles that operate with full 
flight autonomy often become a project unto themselves, with costs and complexities that could be equal 
to or greater than the flight physics research of an X-plane program. 


Complexity Results Summary 


All of the piloting modes shared some complexity topics, such as requirements of a chase aircraft 
during envelope expansion phases, which would include all if not most of the flight-test program. The 
chase aircraft requirement can be complex depending on scheduling, availabiliy, and matching 
capabilities of both aircraft. Other complexity topics that were shared among piloting modes were radio 
communication and power requirements. 

Challenges that were captured for the manned X-plane vehicle were generally minor with relation to 
complexity. Expert observations identified numerous critical (but secondary) research topics that may be 
required before any flight research program can begin for remotely piloted and autonomous vehicle 
options. These secondary research topics were not explored further due to the large scope of potential 
X-plane vehicles. Some topics identified capabilities where remote piloting and autonomous vehicle 
options would be advantageous, but these capabilities did not support the task objectives. One example 
would be an extended mission timeline (longer than 8 hours) where having a pilot onboard could 
introduce prohibitive risk or complexity. For the task objectives, challenges for both the remote piloting 
and autonomous vehicle options increased complexity considerably. The conclusion from the qualitative 
results in this section strongly suggests that a manned X-plane vehicle is likely to be less complex overall 
than the other two piloting modes. 


Qualitative Risk Results 


Topic data were generated that identified supporting rational and challenges with respect to program 
and vehicle risk for each of the proposed types of vehicles. These topic data were then systematically 
scored in order to generate quantitative assessments. 


Manned Vehicle Results Summary 


Manned subsonic vehicles in the weight and class of the proposed vehicles have a long history of 
flight-test processes that are proven and established. Some of the possible new technologies to be 
investigated with the transport of X-planes, such as boundary layer ingestion engines or high aspect ratio 
truss-braced wings, are not common. In these cases a build-up approach to testing can be used to 
gradually push through the unknowns of the program. Other topics that were identified were common 
concerns regarding vehicle or pilot risk in an unknown vehicle configuration and emergency egress. 


Remotely and Autonomous Piloted Vehicle Results Summary 


Remotely piloted and unmanned vehicles were regarded as very similar with respect to program and 
vehicle risk for the following topics. Both types of vehicles would need a flight termination system to 
operate, range safety analysis, and associated mitigations for emergency situations, including loss of 
signal. Risks would include extremely difficult or challenging evaluation of handling qualities, limited 
upset maneuvering capability, and upset recovery testing. One of the most important aspects to these 
types of vehicles is that there is no risk to a pilot, co-pilot, or other required personnel. 

Risks that were identified for remotely piloted vehicles were latency between command and response 
of the vehicle, command and force of the controls, and limiters that could restrict the aircraft from 
necessary recovery inputs in the event of an emergency. 

Risks that were identified for autonomous vehicles were the inability to sufficiently test required 
systems for upset recovery; autopilot, auto throttle, and auto landing. Additionally, approach and landing 
abort logic would need to be integrated. The costs of some novel autonomous vehicles are sometimes 


further increased by necessary redundancies, as they require the construction and maintenance of both the 
aircraft and specialized ground stations as separate entities. 

Common risks to both unmanned configurations were envisioned during the development and testing 
of digital flight control systems, associated human-readable instrumentation, and sense and avoid. There 
was also little confidence expressed in the notion of reusing or adapting ground and flight software from 
the existing medium-sized autonomous fleet. 


Risk Results Summary 


The general capability of the crew of a vehicle to complete sense and avoid detection and action was 
identified as a significant discriminating factor due to EAFB procedures. Most UAS are commonly 
considered to be too small for a pilot, limited to low speeds, inexpensive, and easily replaceable while 
presenting low risk to the uninvolved public. While the complete loss of a small UAS would likely have 
relatively minimal concequences to a program, the same can not be said for medium and large UAS. The 
development and test schedules of larger, faster UAS systems are often extended for a variety of schedule 
and safety reasons. As a best practice, an effect of Instruction 113-1004 and RCC Document 323-99 
(ref. 5), UAS versions of the X-plane aircraft would be considered unproven for at least the first five 
flights. The unproven designation takes into consideration risks to the nearby population, workforce, and 
high value assets. Early flights are often restricted to sanitized airspace on weekend-only (Saturday) days. 
Progression to the experimental risk mitigation level would relieve some of the restrictions in table 1. 
Other issues include cancellations, unforeseen failures, and modifications. Flight-test cancellations are 
common due to weather, instrumentation failures, software, early development problems, and many other 
issues. Modifications such as software updates can be expected to demote or reclassify the vehicle back 
down into the unproven category, requiring the repeat of previous qualification flights. Due to all of the 
dependencies described, it is inevitable that the flight schedules (and flight-test campaigns) of UAS 
X-planes will be impacted and significantly extended. As a consequence of the directions outlined in 
Chapter 14 of EAFB 13-1004 instructions and RCC Document 323-99 (ref. 5), the autonomous functions 
of both types of unmanned aircraft become a significant part of the flight test, which threatens both the 
budget and schedule scope for flight-test research. 


Quantitative Results 


Quantitative values for each topic were assessed in discussions with experts and vetted through group 
consensus. These criteria (table 2) were assigned a value from 0 to 5, with O indicating low and 5 
indicating high complexity or risk. 

For some of the topics, assigning quantitative values was straightforward. For example, the 
implementation of classical flight instruments for a manned vehicle is comparatively simple and therefore 
assigned a quantitative value of one (1). Conversely, assigning quantitative values was quite difficult for 
other topics. Qualitatively, the implementation of environmental control systems for an RPV or 
autonomous vehicle appeared to be a simple evaluation. Environmental control systems for an RPV or 
autonomous vehicle (such as heating, cooling, and weather shielding) were identified as an early 
requirement, but assigning a consensus quantitative score during group review was difficult. 

Summaries of the quantitative results are presented below. The complete list of complexity topics is 
included in Appendix A, while Appendix B contains the list of risk topics. 


Complexity Results 


A summary of the quantitative complexity results is shown in table 5. Applicable complexity topics 
from table 4b were itemized under one of the three categories: VR, RC, and OR. In all of the categories, 
the manned operation mode consistently scored approximately half as complex as either the RPV or 
autonomous modes. The quantitative complexity results numerically support the findings of the 
qualitative results. 


10 


Table 5. Quantitative complexity results. 


Complexity 
Manned Percent RPV Percent Autonomous Percent 

Vehicle requirements (VR) 27 19.4 52 37.4 60 43.2 
22 topics 

Research capability (RC) 21 23.9 31 35.2 36 40.9 
9 topics 

Other requirements (OR) 36 22.0 62 37.8 66 40.2 
17 topics 

Total (48 topics) 84 21.5 145 37.1 162 41.4 


Each of the topics from table 4b were ranked using values from 0 to 5. The complexity scores for 
each topic were separated into three tiers; a score of 0 or 1 was assigned a green bar that indicated low- 
challenge influences, a score of 2 or 3 was assigned a yellow bar that indicated medium challenge, and a 
score of 4 and 5 was assigned a red bar that indicated highly challenging influences. 

Table 6 contains the topics with the largest deviations in complexity. These topics were identified to 
have both low and highly challenging influences depending on which piloting mode is chosen. The only 
topic identified as highly challenging for a manned vehicle when compared to RPV and autonomous was 
the egress topic. Table 6 also indicates that the RPV mode was expected to lie between manned and 
autonomous in terms of complexity. A total of 20 topics were expected to be medium to highly 
challenging for RPV and autonomous vehicles, with autonomous operations expected to be the most 
complex. These complexity results in table 6 further support the findings of the qualitative and 
quantitative results in table 5. 


Table 6. Complexity topics with the largest deviation. 


Complexity RPV Autonomous Total St Dev 
Line of sight 1.25 
Flight termination system 1.41 
Sense and avoid - Pilot SA 1.25 
Stability and control (VR) 1.41 
Egress 1.41 


1.25 
1.70 


Mission management 
Command and control link 


Lost link - Mitigation 1.70 
Cockpit / ground control station 1.25 
Autopilot / auto throttle / auto land 1.70 


Ground operation 
Flight control - conventional 

Mission planning 
Pilot workload quantification 

Handling qualities 


1.25 


1.25 
1.25 
1.63 
1.70 
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Simulation - HILS 1.25 
Envelope expansion 1.25 
Pilot display instrumentation 1.25 
Flight envelope restrictions 1.25 
Operations workforce 1.41 


Each of the topics was then itemized into categories indicating overall low to high challenge, as 
shown in Appendix A. In general, the trend of Appendix A suggests the RPV and autonomous piloting 
modes of operation are expected to be more complex. Appendix A could also be useful for other purposes 
as it identifies low-, medium-, and highly-challenging topics for the overall program independently of the 
piloting mode of operation. 
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Risk Results 


A summary of the quantitative risk results is shown in table 7. Applicable risk criteria from table 2 
were itemized under one of the three categories: VS, DE, and F-TS. In the vehicle subsystems category 
the manned operation mode was estimated to have approximately half the risk of either the RPV or 
autonomous modes. The RPV operation mode risk was evaluated to be significantly less than the 
autonomous operation mode in many individual categories. The manned operation mode was identified to 
contain half the risk of the autonomous modes in many individual categories. The quantitative risk results 
numerically support the findings of the qualitative risk results. 


Table 7. Quantitative risk results. 


Risk 
Manned Percent RPV Percent Autonomous | Percent 

NTIS SUDSY eis OR 27 19.7 55 40.1 55 40.1 

20 topics 
Developmental engineering 

(DE) 22 25.6 28 32.6 36 41.9 
12 topics 

Bllent res suppers eT) 23 23.5 35 35.7 40 40.8 
13 topics 

Total (45 topics) 72 22.4 118 36.8 131 40.8 


Each of the risk topics were also ranked using values from 0 to 5. The scores were separated as 
described in the Complexity Results section. 

Table 8 presents the risk challenges that showed a strong dependency with the piloting mode of 
operation. Nine topics were expected to be more challenging for RPV and autonomous vehicles, again 
showing a trend that each of these piloting modes of operation are expected to be more difficult. The 
complexity results in table 8 further support the findings of the qualitative results and numerical results. 


Table 8. Risk topics with the largest deviation. 


Risk 


Manned RPV Result 


St Dev 
1.25 
Airframe performance 7 1.25 
Pilot situational awareness 8 1.25 
Command and control link 8 1.25 
Cockpit / ground control station 9 1.63 
Autopilot / auto throttle / auto land 9 1.41 


Autonomous 


Lost link 


Airworthiness process 


Flight control - Conventional 


Handling qualities 


Mission timeline 


1.25 
1.70 
1.70 
1.70 


As stated in the Complexity Results section, each of the topics was itemized into categories indicating 
overall low to high risk, and are included in Appendix B. In general, there is a strong trend in table 8 that 
suggests the RPV and autonomous piloting modes of operation are expected to have more risk. 
Appendix B could also be useful for other purposes, as it identifies low-, medium-, and high-risk topics 
for the overall program independently of the piloting mode of operation. An attempt to time weight the 
results in table 5 and table 7 produced small differences in the overall results and is not presented. 
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Conclusion 


The aircraft studied during the CAS X-plane subproject were chosen to demonstrate new approaches 
for revolutionary increases in sustainable air transportation flight efficiency. All of the CAS X-plane 
study vehicles considered were medium-sized vehicles, approximately the size of a G-III aircraft, with the 
ability to sustain flight conditions appropriate for subsonic and transonic air transport research operations. 
Analysis assumptions were summarized in table 3. Additionally, the X-plane concept vehicles were 
assumed to be subsonic medium-scale aircraft. In a program of the projected scale, the loss of vehicle was 
anticipated to be equally detrimental in terms of risk as loss of a pilot; it was assumed that either would 
cause the program to be canceled. 

Expert group evaluations provided qualitative reasoning and substance from a diverse set of pilots, 
engineers, and other experts on the many issues surrounding the decisions about piloting options. The 
group evaluations were estimated within each topic to provide a quantitative result that summarizes and 
supports the qualitative results. 

An EAFB instruction document was identified that, when implemented, mitigates risk by requiring a 
new or low flight number vehicle to systematically perform sorties and pass numerous review boards 
during the testing of the vehicle before being granted further operational flexibilities. The EAFB 
document clearly describes the accepted maturation process and specifies the extra restrictions applied to 
unmanned vehicles throughout the various developmental and testing stages. The EAFB instruction 
13-1004 as well as the implications of its implementation emerged as the source of the effects found in 
the qualitative and quantitative data. 

Overall, this study concluded that a manned aircraft option could be expected to suppress complexity 
and risk for a potential X-plane program as defined in this document. The reduction of these factors was 
expected to enhance flight research efficiency and constrain the cost of the flight-test portion of the 
program. Autonomous functions in UAS vehicles have a way of becoming a large part of the test 
program, which would likely distract from the fundamental flight experiment and/or increase costs and 
schedule. In demonstration of the primary research objective, air transportation flight physics, both 
remotely piloted and fully autonomous vehicles are likely to increase both cost and schedule compared to 
conventional manned operations. The study showed both qualitatively and quantitatively that a manned 
aircraft was the best choice to align with test activities for transport aircraft flight research from a reduced 
complexity and low risk perspective. 
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Appendix A 
Summarized Results for all Complexity Topics 


Complexity Autonomous Total St Dev Result 
Lost link - self-destruct 2 0.47 
Radar 3 0 
Transponder 3 0 
Power requirements 3 0 
Radio communication 3 0 
Ground test - taxi, et cetera 3 3 7 0.94 
Navigation 8) 3 7 0.94 
Range restrictions 3 3) 7 0.94 
Sense and avoid - airspace 3 3 7 0.94 
availability 
Frequency requirements 5 5 7 0.94 
Flight instrumentation and data > > > 6 0 
seams Medium overall 
Chase aircraft 2 es es 6 0 : 
Laminar flow 2 3 3) 8 0.47 pon ensy 
Engine performance 3 5) 5) 9 0 
Environmental control system m2 ve 8 0.94 
Stability and control (RC) A 9 0.82 
Simulation - 6-DOF 9 0.82 
Control law development 9 0.82 
Ground tests - GVT, et cetera 10 0.82 
Mission timeline 10 0.94 
Airframe noise 10 0.94 
Airframe performance 10 0.94 
Control room 10 0.94 
Mission support 10 0.94 
Flight control - Full DFCS 12 0.82 
Engineering workforce 13 0.94 
Flutter 14 0.47 
Airworthiness process 15 0 
Line of sight 5 1.25 
Flight termination system 6 1.41 
Sense and avoid - pilot SA 6 1.25 
Stability and control (VR) 6 1.41 
Egress 6 1.41 
Mission management 7 1.25 
Command and control link 7 1.70 
Lost link - mitigation 7 1.70 
Cockpit / ground control station 8 1.25 
Autopilot / auto throttle / auto land 8 1.70 Largest deviation 
Ground operation 8 1.25 in complexity 
Flight control - conventional 8 1.25 
Mission planning 8 1.25 
Pilot workload quantification 9 1.63 
Handling qualities 10 1.70 
Simulation - HILS 10 1.25 
Envelope expansion 11 1.25 
Pilot display instrumentation 11 1.25 
Flight envelope restrictions 11 1.25 
Operations workforce 12 1.41 
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Appendix B 


Summarized Results for all Risk Topics 


Risk Manned RPV Autonomous | Result St Dev Result 
Laminar flow 3 0 
Power requirements 3 0 
Radio communication 3 0 
Transponder 3 0 
Flutter 2 4 0.47 
Stability and control - evaluation 3 5 0.94 
Egress 3 5 0.94 
Line of sight 2 3 3 0.47 
Chase aircraft 2 2 5 0.47 
Range restrictions 2 2 5 0.47 
Radar 2 3 6 0.82 
Mission management 2 3 6 0.82 
Frequency requirements 8) 2 6 0.82 
Navigation 6) 3 7 0.94 
Pilot display instrumentation ) 3 7 0.94 
Flight termination system 3 3 7 0.94 
Control law development 2) 5) 7 0.94 
Engineering workforce - development & a es 6 0 
Simulation - 6-DOF 2 2 2 6 0 
Flight instrumentation and data 2 2 2 6 0 
recording 
Ground tests - preflight, GVT, et cetera & a 2 6 0 
Instrumentation ground testing 3 2 Pa 6 0 Medium 
Operations workforce - physical danger 3 2D Bs 7 0.47 evel nke 
Engineering workforce - test support 4 3 5 8 0.47 
Simulation - HILS & 3 3 8 0.47 
Operations workforce - program a 3 3 8 0.47 
Environmental control system ® 3 I 9 0 
Engine performance 6) 8 a 9 0 
Airspace availability 2 6) 9 0.82 
Stability and control - development 2 é) 9 0.82 
Envelope expansion &, 3 9 0.82 
Upset recoveries ys, S 9 0.82 
Flight control - full DFCS 2 10 0.94 
Mission support 2; 10 0.94 
Ground Testing - taxi, et cetera & 10 0.94 
Lost link 2 7 1.25 
Airframe performance 2 7 1.25 
Pilot situational awareness 2 8 1.25 
Command and control link 2 8 1.25 
Cockpit / ground control station 3 9 1.63 | 
: deviation in 
Autopilot / auto throttle / auto land 9 1.41 ilk 
Airworthiness process Bs 8 10 1.25 
Flight control - conventional 10 1.70 
Handling qualities 10 1.70 
Mission timeline 10 1.70 
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